Skip to content

SEA: Decouple Link Fetching #632

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 368 commits into from
Jul 21, 2025
Merged

Conversation

varun-edachali-dbx
Copy link
Collaborator

@varun-edachali-dbx varun-edachali-dbx commented Jul 9, 2025

What type of PR is this?

  • Feature

Description

For efficiency, we fetch the links for data in the SeaCloudFetchQueue in a separate thread. Link fetches are slow because they come from the bottlenecked control plane, so coupling the fetching logic with the main process slows down our fetch phase. Instead, we proactively fetch links in the background and use them when required.

Note: this PR does not account for link expiry. This will be handled separately in #635.

How is this tested?

  • Unit tests
  • E2E Tests
  • Manually
  • N/A

Related Tickets & Documents

N/A

Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
@databricks databricks deleted a comment from github-actions bot Jul 20, 2025
@databricks databricks deleted a comment from github-actions bot Jul 20, 2025
@databricks databricks deleted a comment from github-actions bot Jul 20, 2025
@databricks databricks deleted a comment from github-actions bot Jul 20, 2025
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
@databricks databricks deleted a comment from github-actions bot Jul 21, 2025
@databricks databricks deleted a comment from github-actions bot Jul 21, 2025
@varun-edachali-dbx varun-edachali-dbx merged commit 806e5f5 into sea-migration Jul 21, 2025
22 of 23 checks passed
varun-edachali-dbx added a commit that referenced this pull request Jul 22, 2025
varun-edachali-dbx added a commit that referenced this pull request Jul 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants